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REMARKS 

Claims 8-11, 16-19, 22, 23, 26 and 27 stand rejected under 35 U.S.C. §102(b) as being 
anticipated by Turtle et al. Claims 12- 14, 20, 21, 24, 25, and 28-36 stand rejected under 35 U.S.C. 
§ 103(a) as being unpatentable over Turtle et al in view of Perholtz et al. The Examiner's 
consideration of the present application is most appreciated. In view of the present amendment and 
following remarks, the Examiner is respectfully requested to reconsider the outstanding rejection and 
allow the present application to issue. 

Interview Summary 

Examiner Heffington is thanked for the personal interviews throughout this examination, 
including those granted March 22 and May 4, 2010. The Interview Summaries provided by Sara 
England in regard thereto are accurate and complete. No agreement was reached. 

The Present Amendment 

By way of the present amendment, the claims now more clearly recite what the applicants 
regard as the invention. Each of the independent claims have been amended herein to specifically 
recite searching for an element or entity which may be found at an indeterminate location within the 
system under test. Support for this language is found on page 8 in lines 17-19, between page 10 line 
4 and page 1 1, line 9, and between page 15, line 4 and page 16, line 10 of the present specification. 

Newly added independent claim 38 incorporates language similar to claim 8, with fewer 
limitations in new claim 38. Newly added claims 40 and 42 incorporate language similar to claim 28, 
both slightly reworded. Finally, dependent claims 43-77 incorporate features that were discussed in 
the May 4, 2010 interview as potentially adding novelty to the present invention. The Examiner's 
consideration of this new claim language is respectfully requested. 

In addition, and arising from the foregoing interviews, the independent claims have been 
amended to recite "automatic", "by a script", and other similar phrases where necessary to clearly 
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delineate that the present invention is in fact a machine, and not abstract ideas of a person. Support 
for these phrases may be found, for exemplary purposes, between page 10 line 21 and page 1 1, line 
11. 

With regard to discussion of action occurring that changes when found or not found, claim 
14 in line 19 has the phrase "result of inserted, to now read: "said flow which varies responsive to 
a result of detection". The remaining claims already incorporated this limitation. In addition, claim 
28 has been amended herein at the very end to state: "a first user input command implemented 
responsive to a detection of said first entity during said image processing step and a second user input 
command implemented responsive to a non-detection of said first entity during said image processing 
step." 

Support for the newly added claim language may be found in several places. From page 8, 
line 14 - page 9, line 1 : "More particularly, window 1 10 provides a command capture interface which 
most preferably provides a system-under-test (SUT) screen display 114 which provides a visual 
depiction of the display screen generated by the system for which the operation of the interactive 
development environment 100 is designed to control. Within that display there will be various 
graphical representations 116 that may take the form of icons as illustrated, or various buttons, 
message boxes, prompts or any other graphical image or control. The depiction may be an exact 
copy of the remote graphical user interface, a resized or scaled version, or may further include various 
image manipulations such as color conversions or other manipulations as deemed suitable and 
appropriate for a given application." 

Further support is found on page 10, lines 4 - 15: "Where a single entity is anticipated, the 
image of the entity can be stored through the command capture interface window 110, and, when 
such image is later detected during execution of a script, an appropriate action can be selected, such 
as the "Click" command which would represent a mouse click upon a particular point in the screen. 
While many times this "Click" command may be executed directly upon the entity which newly 
appears upon the system-under-test screen 1 14, the direct action upon such an entity is not required. 
Instead the user of interactive development environment 100 has complete control over any of the 
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user actions that may be relayed to the system-under-test 290, such as providing typed text, 
commands, movement of the mouse, and so forth. Consequently, the appearance of an object may 
stimulate any suitable action." 

Additional support is found between page 10, line 21 and page 11, line 11: "These image 
commands will most preferably include screen image searching commands and specific image 
information. Exemplary of the screen image searching commands are such commands as "WaitFor", 
"WaitForAny", "WaitForAU", "RefreshScreen", "ImageFound()", "AnyImageFound()", 
"ImageLocationO", ' AnyImageLocation()", "EveryImageLocation()", and other similar commands. 
A variety of information will most preferably be obtained or obtainable with regard to specific images, 
through such commands as "ImageInfo()", "FoundImageNumber()", "FoundImageName()", 
"FoundlmageLocationO" "ImageHotSpotO", and "ImageSizeO". Utilizing the above command set, 
it is possible to monitor a graphical user interface for any type or shape of image and then, responsive 
to the presence thereof, select a subsequent user action as though the user action were being 
performed directly upon the system-under-test 290 rather than from a source or controlling computer. 
In the event an unexpected event or entity appears upon the screen, the user, through the integrated 
development environment 100, has the opportunity to control the operation of the local and remote 
systems responsive thereto." 

From page 14, lines 9-11: "Communications channel 275 will in the preferred embodiment 
include a keyboard channel 272, mouse channel 274, and a transfer of screen updates from VNC 
server 280 back to the remote GUI interface 270. Communications channel 275 may be a high speed 
trunk line or cable, or may alternatively be a relatively slow-speed dial-up or RS-232 type connection. 
With proper selection of components, the preferred embodiment has much flexibility to operate 
through diverse communications channels having very different data transfer rates and signal to noise 
ratios." 

From page 15, lines 1-16: "GUI recognition subsystem 265 dynamically scans the screen 
image of the remote sy stem-under- test 290 for any bit-map images which the initiate action subsystem 
260 is searching for. The goal of GUI recognition subsystem 265 is to locate images and signal the 
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presence of such images to the initiate action subsystem 260 through an image detection signal, 
initiate recovery procedures such as moving the mouse cursor to locate images, or to report that the 
sought-after images are not available. GUI recognition subsystem 265 cooperates with the initiate 
action subsystem 260 and language extensions processor 255 output to determine whether a desired 
event or image has been created and if so, to execute the desired image identification actions 
requested within the language extensions received from language extension processor 255 through 
the execution environment 245. Initiate action subsystem 260 initiates any action requested by the 
language extensions received from language extensions processor 255, which could be active 
commands to type text through the keyboard or to move the mouse in a number of ways. The 
commands may also in the preferred embodiment include passive commands to search the screen 
image on a continuing basis for a specific bit-map image, and, for exemplary purposes, terminating 
the search after a given period of time has elapsed." 

If there remain any specific recitations that the Examiner is uncertain about the support, the applicants 
and their representative will endeavor to provide specific citations. 

Exemplary Specific Novel Features Found in the Present Claim Recitations 

The present invention searches the pixel image of a machine under test for "a first graphical 
element which may be found at an indeterminate location contained within and comprising less than 
said pixel image" (claim 8, lines 6-9). Similar recitations are found in the remaining independent 
claims 14, 16, 28, 38, 40, and 42. The individual elements may comprise "icons as illustrated, or 
various buttons, message boxes, prompts or any other graphical image or control" (Present 
specification, page 8, lines 17-19). The Webster's Unabridged definition for "search" is attached 
herewith for review and consideration by the Examiner. Therein, "search" in verb form is defined as 
"1. To look over or through, for the purpose of finding something." As a noun, "search" is defined 
as "1. An act of searching; scrutiny, inquiry, or examination in an attempt to find something, gain 
knowledge, etc." This clearly distinguishes from Turtle, where a comparison, not a search, is the 
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only action disclosed. 

In addition to this search for the first graphical element, which might be found at different 
locations upon the graphical user interface, the present invention further recites the step of: 
"generating a user peripheral input device input action within said second computer system graphical 
user interface . . . responsive to said receiving and results of said searching steps." (claim 8, lines 1 1- 
16). Similar but alternative language is found in claim 16, lines 1 1-16, and claim 14, lines 18-24. 

Claim 24 additionally recites: "locating a user peripheral input device input action at a location 
relative to said first entity" (claim 24, lines 3-4). Similar language is found, for example, in claim 29. 

As a result of these features, the user does not need to worry where on the screen the image 
element appears, and can make intelligent decisions about any subsequent actions, including the 
placement of a mouse click or other similar action relative to the placement of the individual element, 
and also alternatives that might arise and still be handled to continue the testing. 

By being able to search for image elements and make intelligent decisions, a calculator can 
be tested through all numbers and operations by storing the images for each of the operations and the 
numbers 0 - 9. For a simple calculator, the operations might be "add", "subtract", "multiply", 
"divide", "equals", and "clear". Then, using simple ""For Next", "Repeat With", "Do While" loops 
or the like, each of the operations may be tested and the results confirmed by saving the images of 
operations, numerals and any other necessary elements. In the present invention, this for exemplary 
purposes might be achieved using very simple loops requiring less than forty lines of code. Both the 
code and the images will take up very little storage space. This is done without concern for where 
on the screen the calculator might appear. In contrast, Tuttle must actually execute every possible 
test, and then save the image to compare with future systems. The storage of these Tuttle images will 
require an impractical amount of time, and an unreasonable amount of storage space. Even the 
identification of the digits of the most common calculator, an eight digit calculator, would require 100 
million Tuttle comparisons to match the digits alone, and, again, any window cascading or 
repositioning would cause a failure in the Tuttle comparison. 
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The Turtle Reference 

The Examiner will recognize that Tuttle et al stores a representation of the entire screen, or 
a predetermined portion of lines or screen width, and then checks during testing to see if the exact 
pattern reproduces later. Unfortunately, a large number of factors will change between diverse 
computers that will interfere with this test. Tuttle contemplates some of these factors, such as 
flashing cursors, and considers cutting off a portion of the screen to remove clocks and the like. 
However, the Examiner will recognize, for example, that modern windowing programs normally 
cascade, meaning that when a new window is opened, the new window will be shifted down and to 
the right from the previous window. So, unless exactly the right number of windows have been 
opened prior to the start of the Tuttle testing system, Tuttle will reject a perfectly fine operation due 
to the shifting resulting from cascading windows. In other words, Tuttle only discloses a comparison 
of screens or screen signatures, without any searching within the screen for elements or entities at 
indeterminate locations. 

In addition, since Tuttle is only able to match a screen for "pass/fail regression testing" (Tuttle 
Col 14, lines 38-41), there is no opportunity to choose what action to take, depending upon what 
occurs. Instead, Tuttle must simply halt the program and return an error when a screen image fails 
to match. 

The Examiner will appreciate that Tuttle is consequently limited solely to testing of hardware 
such as might occur at a manufacturing facility, where every computer is loaded with the same 
software, and is tested immediately at start-up so that subsequent windows will cascade at the proper 
location. 

Tuttle fails to illustrate or teach the detection of a graphical element as recited, and does not 
respond to either control execution flow or to control passage of a signal through an i/o channel. The 
remaining references of record fail to rectify this deficiency of the prior art. Consequently, the 
anticipation and obviousness are herein traversed, and the Examiner is respectfully requested to 
withdraw this basis for rejection. 

No new matter has been added by way of the present amendments. Consequently, in view 
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of the present amendment and remarks, the Examiner is respectfully requested to reconsider the 
rejections of record and allow the present application to issue. The present response is not considered 
to acquiesce with regard to the novelty of the dependent claims not specifically addressed herein, and 
should the Examiner need to consider these as well, the Examiner is also respectfully requested to 
specifically reconsider the novelty found therein in light of the foregoing discussions. 

Should there remain any open issues in this application which might be resolved by telephone, 
the Examiner is respectfully requested to call the undersigned at 320-363-7296 to further discuss the 
advancement of this application. 

Sincerely, 

/Albert W. Watkins, reg. 31,676/ 
Albert W. Watkins 
reg. 31,676 



